I have a single multi-role Exchange 2013 server and it would appear that it's not properly maintaining the temp files for the transport service.  I still have all those folder locations at their default and the problem folder is c:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\data\Temp

I never had a problem with this in Exchange 2007 but I am used to running a PowerShell script nightly to clean up the IIS log files.  Do I need to do something similar for this temp folder?  Is there a setting I can adjust so that Exchange will limit the size of this folder itself?  If I stop the transport service and delete the files here will I lose anything?

Any suggestions or insight would be greatly appreciated.

• Edited by Gaven LS Thursday, August 08, 2013 5:48 PM

Hello,

Transportrole uses and holds many temporary files during the course of a mail message. Most of these get overwritten with new ones. So if some files haven't been used for long of a period, you can safely to remove them. But as with anything server related, deleting a file can have unpredictable results later.

So I remommend you to move them off to another location or removable media. 

I don't recommend you to running a powershell script to clean up the temp folder.

If you want to delete the temp files, I recommend you to back the temp files up to avoid unpredictable results.

If you have any feedback on our support, please click here

How big is the temp folder? It shouldn't be growing uncontrollably.

It got up to about 17 GB and then back pressure kicked in and stopped accepting messages.

Hello,

Do you try to move them off to another location or removable media?

Besides, I recommend you to use server health and performance to check your exchange server. 

If you have any feedback on our support, please click here

Hello,

Is there any update?

If you have any feedback on our support, please click here

I would ensure that the AV is not scanning this folder, make sure that you have appropriate exclusions set in place. the content conversion process takes place on this location & its important to make sure that antivirus is not scanning this location. this could be causing the files not to be automatically removed due to AV Lock

I have the same problem!

And I've learned that folder filled with files corresponding to the attached file is sent out of the organization. Attachments sent internally does not end in the folds.

I've submitted a request to have the folder excluded from SEP 12.  I'll post back if that solves the problem.

I have tryed to uninstall the SEP 12, and that did not solve the problem!

Same - I had that folder excluded from SEP and they are still building up.

I have the same problem and at the moment I got ~20GB of files in TEMP folder and it's still growing. I wonder why does it happen and is this normal? And can I safely delete content of this folder? Has anybody already tried to delete it ?

• Edited by przemyslawb Tuesday, October 15, 2013 1:11 PM

I have removed the files that are more than a week old. It did not cause any problems.

Has anyone opened a case with Microsoft on this? I have been running exchange since 5.5 and I have never seen an issue like this. We moved to Exchange 2013 and in the three months it has been in production it has crashed from this bug five times!!!

Microsoft has got to fix this issue!

Just so everyone knows, I have opened a case with Microsoft on this.

I expect the answer to your case , i have the probem with my exchange 2013.

As some additional information, it happened again this morning to me. What I found is that the mail.que file had grown to 38,805,568 KB in size. At that point, Exchange did something and moved the mail.que database to a folder named "Messaging.old-20140129135432" which causes the transport service to crash. I have pulled the "corrupted" mail.que database off the server and I am going to upload it to Microsoft for their review.

I will let everyone know as I get more information.

Thanks for following up Will. We are seeing this as well, we might get 1GB over a week in the temp directory. I'd be interested to see how you go with this. Happy to try and help if there is anything you'd like tested in a different environment, assuming we actually have the same issue. I did find an old 2007 article that references the directory (http://technet.microsoft.com/en-us/library/bb738141(v=exchg.80).aspx) which indicates it might use this directory when under memory pressure, not sure if that has anything to do with this.

Peter

I heard back from Microsoft today and this was their response:

From your description, I know you have copied the 38GB mail queue to another server to do a database cleanup. But it seems the database is dirty shutdown with one log missing. As the log is already missing, if theres no backup for the log, we can only use eseutil /p to make the database into clean shutdown. But this command will cause some data lose. See more details in the following link: http://technet.microsoft.com/en-us/library/aa997215(v=exchg.65).aspx.

On the other hand, I know after rebuilding the mail.que file, the file is still growing fast. Here , I would like to say that it is a normal behavior in exchange 2013. As far as mail.que file is concerned, we cannot compare exchange 2013 with exchange 2007 or 2010. Exchange 2010 or 2007 do not have this feature called SafetyNet. SafetyNet feature is the one which is responsible for the large size of the mail.que in Exchange 2013.

• In Exchange 2013, transport high availability is more than just a best effort for message redundancy. Exchange 2013 attempts to guarantee message redundancy. Because of this, you can't specify a maximum size limit for Safety Net. You can only specify how long Safety Net stores messages before they're automatically deleted.
• The length of time successfully processed primary messages are stored in Primary Safety Net, and acknowledged shadow messages are stored in Shadow Safety Net. Unacknowledged shadow messages eventually expire from Shadow Safety Net
• So also need to consider the point that depending change in the number of messages or amount of mail flow on an any given day is also going to impact the size of mail.que for an extended period of time as compared to exchange 2007 or exchange 2010.

See more details in the following link: http://blogs.technet.com/b/exchange/archive/2013/05/06/ask-the-perf-guy-sizing-exchange-2013-deployments.aspx

Given the situation, we can set the SafetyNetHoldTime to a little value such as 5 minutes to see if the mail.que is still large. See more details in the following link: http://technet.microsoft.com/en-us/library/jj657495.aspx

In addition, I find you want to change the retry value. In General, we can change it via QueueGlitchRetryCount and QueueGlitchRetryInterval. See more details in the section Configure the transient failure retry attempts, the transient failure retry interval, and the outbound connection failure retry interval in the following link: http://technet.microsoft.com/en-us/library/aa998043(v=exchg.150).aspx.

In addition, I have created a workspace for you, you can upload the information to this workspace: [Workspace information:]

1. You will receive another Email named from, please check your Inbox and Junk Emails.
2. Please download the Microsoft Data Transfer and Management tool (DTM.exe) and install it, then you can use temporarily logon information above Email has mentioned to logon the DTM tool.
3. You will be requested to change the temporarily password when you first logon. If you forget the password you have changed, please check the following link to reset the password. https://filetransfer.support.microsoft.com/EFTClient/Account/LostPassword.htm
4. Once you logged on the DTM tool, please upload the related data to us. Please see above information and if theres anything unclear, feel free to contact us.

Now, I checked my SafetyNetHoldTime and it was set to 2 days (the default) as is my ShadowMessageAutoDiscardInterval. That totals processed mail being held for up to four days. I do not think this explains why the mail.que database continually grows up to a certain point and then crashes.

I have another theory on why this is happening: I do not think that the jet engine powering the mail.que is recovering its whitespace as it should. I have tried to test this theory but running eseutil /p against my 38GB "corrupt" mail.que database to bring it back to a clean state so that I can defrag it but the /p has been unsuccessful to this point.

I am uploading the entire 38GB "corrupt" mail.que and its log files to Microsoft for analysis. Hopefully, they will have a better solution than for me to turn down my SafetyNetHoldT

For what it's worth my mail.que is sitting comfortably at 7.75 GB and does not appear to be growing out of control.  However, the Temp directory continues to grow.

Last night my server (with CU2) locked up because the C drive was out of space. After looking around I found 12 GB of files in the "V15\TransportRoles\data\Temp" that were dating back to when I first built this server. Lucky for us that this server is a VM, so we were able to increase the HD size by another 20 GB till we can get this sorted out. I do run a batch file everyday to delete log files older than 3 days in some of the folders that I know have been cleared by MS (from a previous support ticket I had with them), but I did not include this one since they never mentioned it.

For those looking to get rid of logs (as long as you have a successful backup), then you can use Task Scheduler to run the following from the command line:

forfiles.exe /p "c:\inetpub\logs\LogFiles" /s /m *.log /d -2 /c "cmd /c del @file"
pause
forfiles.exe /p "c:\Program Files\Microsoft\Exchange Server\V15\Logging" /s /m *.log /d -2 /c "cmd /c del @file"
pause
forfiles.exe /p "c:\Program Files\Microsoft\Exchange Server\V15\Logging\Diagnostics\DailyPerformanceLogs" /s /m *.* /d -2 /c "cmd /c del @file"

My mail.que is fine and sitting around 5.5 GB, but the "transportroles\data\temp" directory grows everyday. I will also be opening a ticket with MS Support regarding this today. If MS Support says I can include the "V15\TransportRoles\data\Temp" directory then I will add that to my batch file.

Any update for this case please?
I faced this problem too.

Any update for this case please?
 I no need to delete temp file without the root cause of this problem

nothing update please? :(

I thought the problem was with the Temp directory not the mail.que file? Our mail.que file is fine. We are also just deleting older files out of the temp directory to work around the issue. A problem with the mail.que file growing could be caused by quite a lot of different things, and I think likely to be unrelated to the temp directory issue.

Supawat, we still do not have a root cause on the files in the temp directory. I can confirm we are automatically deleting them when they get to 14 days old, and have been doing so for about a month or so without any impacts that we have noticed. I know that probably isn't much comfort for you. I'd suggest raising a case with MS if this is important/urgent for you, if you have support. We have not raised a case with MS yet as we have higher priority issues to resolve first, when I get a chance to raise a case on this I will post the results here.

Peter

Here is the latest from Microsoft Support:

Hello Will,

I tried calling you on phone numbers (xxx) xxx-xxx, (xxx) xxx-xxx to discuss on this case and reached your voice message.

Regarding the mail.que growing, as you mentioned it is normal in exchange 2013 server. You can have adequate disk space on the drive where the queue is located. http://blogs.technet.com/b/exchange/archive/2013/05/06/ask-the-perf-guy-sizing-exchange-2013-deployments.aspx

Once the queue database grows to an extent when the transport unable to process it, the mail.que will get renamed and new queue database will be created automatically. Then you can delete the old queue database manually to free up the disk space.

You also wanted to transfer the case to development team, but we cannot involve them on this case since it is by-design behavior (http://support.microsoft.com/gp/proffaq/zh-tw). Please let me know your convenient time and phone number to discuss on this.

Thanks and Regards,
Dinesh

Needless to say, I was furious with this response! Here is what I sent back to him:

Dinesh,
This is the first time any Microsoft engineer has said this behavior is by design! Not only that, this by-design behavior is flawed for several reasons:

• There is no guidance on how much disk space will be needed or how to calculate the needed space.
• In my case, it moves the mail.que but does not build a new one so mail flow is interrupted.
• It does not move the old mail.que log files so that even if a new mail.que is built, the Transport service cannot start because the mail.que database and the log files are out of in sync and the log files must be removed!

Now, the link to the TechNet blog article you sent discusses sizing for the transaction logs, but does not mention sizing for the mail.que database or how it will wildly grow out of control! So how can any Exchange Administrator properly size their installation?

Now, I have been supporting Exchange since 5.5 and in environments much larger than this (20,000+ mailboxes across multiple countries and continents) and you cannot tell me that this behavior is by design. I have worked too long with Microsoft and Microsoft support to agree with your stance that this behavior is by design! Additionally, if this behavior was by design, dont you think in all the Technet support forum posts that some Microsoft MVP or support engineer would have mentioned that!

Dinesh, I have copied my account rep (XXXXXXX) on this e-mail and I will be posting the response you sent me below to the Technet forum posts as well. Please make sure that you understand my issue and the behavior we are experiencing and then please involve a more senior engineer OR transfer it to the development team as I requested!

I have already spoken to my account rep and he is escalating this!

By design my b

Hi Will..

Is the growth in your mail.que a result of messages being stuck in the queue, i.e. a down destination domain or something? I'm sure you've already checked, but thought it might be worth throwing out there. Get-TransportService | Get-Queue | Measure-Object MessageCount should give you an idea, just in case.

We sometimes see blowout on our mail.que files when a mass mail is sent with a large attachment (we have a 50MB attachment limit, and more than 80,000 users), might be worth checking your message tracking logs to see if there was a big run of large messages that might have grown it.

I've never seen a mail.que be renamed, but then we have never really hit severe disk pressure on the volume hosting it either, it might well be a protection mechanism in the product, I'd be asking premier for documentation on this feature if it is by design as they say it is.

Are you still getting the temp directory fill with other files, or was that just the renamed mail.que?

Regards,

Peter

mail.que is a red herring! The problem is the build up of files in the Temp directory identified dozens of times in this thread.

Very appreciate for that :)

Thank you for your answer.

Sorry Gaven, completely missed that you raised this question. We were suffering from memory pressure, and have recently increased the memory on our servers, so it will be interesting to see if our temp directory usage has changed at all given the reference here: http://technet.microsoft.com/en-us/library/bb738141(v=exchg.80).aspx. I'm not holding my breath as that is for 2007 Edge role, but you never know. I'll do some more analysis and see if I come up with anything, otherwise I might raise a MS ticket.

OK. I raised a ticket on this. They know about it. Apparently it occurs more on Windows Server 2012 than Server 2008 R2. I closed my ticket without going down the rabbit hole of trying to get it fixed, but I assume it will be on the radar to be fixed at some point. It didn't sound like anyone had pushed for them to actually chase root cause or to get it fixed, so that might not be the case.

So the directory is used for content conversion in the transport pipeline, which we had already assumed anyway, but the outcome was the files are safe to delete even if the message is still in the transport queue. In fact, I was told if you can delete the file (i.e. if it isn't locked), it is safe to delete.

I am going to err on the side of caution and leave our script to delete them after 14 days without modification.

Hope that helps

Thanks for posting this.  I will likely be doing the same.

Also, we are at just under 60GB of temporary data at the moment.  I imagine this would break many servers.  

Get it together Microsoft!

We have the same problem ... 4 Exchange 2013 Servers with one DAG and one of these servers have a size of 28GB in C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\data\Temp

:-(

I just had a call with Microsoft and escalated this is a high priority case. I got a call back from an engineer and she told me that the filling of this folder is caused by the malware agent.

Quote:

As mentioned in our phone conversation, this is a known issue for which a fix is to be released in CU5. CU5 is scheduled for the middle of June 2014.

As a workaround we suggest to disable the Malware Agent on the Exchange 2013 server and/or delete the files for the last two weeks in the folder TransportRoles/Data/Temp

Unquote

I just left the malware agent running and deleted everything in that folder older than 2 weeks, no problems and got 20 GB per server back

Regards

Danny

Thanks for sharing

Cumulative Update 5 for Exchange Server 2013

http://support.microsoft.com/kb/2936880/

Anyone tried to install and confirm if it fix our problem?Thanks,

Riccardo

I just installed CU5 today and I found this bug :(

Exchange 2013 CU5 , Exchange Power Shell very very very slow reasponse when using get command.

http://social.technet.microsoft.com/Forums/en-US/54f37cf2-5c9d-4c64-81ef-c09cc400deac/exchange-2013-cu5-exchange-power-shell-very-very-very-slow-reasponse-when-using-get-command?forum=exchangesvrdeploy

Grey hair at all

CU5 appears to fix the issue of the temp directly filling up. However there is a massive gotcha with applying the update where it fails if there are receive connectors with bindings or remote IP ranges that overlap such as a manually created internet receive connector.    This configuration was possible in previous Exchange releases but no longer after CU5.    The server that I updated had to be recovered using setup /m:recoverserver after deleting the offending receive connector using ADSIEdit

Mark

Installed CU6 and it fixed the problem with the temp directory filling up.

Looking over the list of problems CAUSED by each cumulative update gives me little hope for stability in Microsoft's future.  

I can't imagine we will ever want to upgrade to a newer version of Exchange again until it has been on the market for several years.  

CU5 Should fix the

Anyone with an update ?

I've have installed CU7, and it still updates the temp folder.
every minutes it incresses with a new file aprox: 30 MB in size.

--------------
Jan

Adding my name to the list of those needing / wanting an update. Mohameds response says 'Should'

Can anyone actually Confirm? We're up to CU7 now

hmmmm I just installed Cu7 and it seems the temp folder stopped filling on my server. It was driving me insane trying to figure out what was filling this drive besides logs from the web server and the Window Event logs for Exchange... Thanks for posting this thread as well since I wasn't sure what to do with this Temp folder contents. Hopefully the last few CU's have resolved this stupid bug.

Did CU7 delete all the "very old" temp files that perhaps had accumulated over many months ?

Or did it just know how to "stop adding 'new' additional temp files that are actually not needed anymore" ?

Thanks.

Hi Tycho-1

We have multiple Server 2012R2 + Exchange 2013 SP1 machines for ourselves and our clients.

We came across this issue with one and found this article.

We upgraded to CU5, it appears that it does not clear the old temp files in the temp directory, but we haven't had anything added to that directory since 2014, today we just deleted 20GB of files from the temp directory that are last modified in 2014.

We have a CU8 Server and will be upgrading all other to CU8 soon, ill try and report back with our findings in the future.

For now, I would say upgrade to CU8 or what ever is the latest CU when you are reading this, and We deleted ALL files from that temp directory and it had no adverse effects on the server.

We rebooted straight after the deletion.